A motion of a mobile body, such as a vehicle, a mobile robot or the like, is usually calculated by any of two methods: a method of using a sensor which detects a rotation of a wheel, an angle of a steered wheel, an acceleration, an angular acceleration or the like; and a method of using images of a surrounding of a mobile body which are captured by a camera mounted on the mobile body.
A method of using a sensor which detects a rotation of a wheel or an angle of a steered wheel is advantageously stable in because of a small error and less frequent failure of detection in certain circumstances, such as a flat and dry road surface or the like. However, this technique has a problem that a large error in principle occurs in circumstances, such as a slippery road surface or wheel, an uneven road surface or the like.
A method of using a sensor which detects an acceleration or an angular acceleration can be used to calculate a motion of a mobile body in circumstances, such as a slippery road surface or wheel, an uneven road surface or the like. However, this method has a problem that a large error occurs, for example, if a ratio of an acceleration or an angular acceleration caused by a motion or an attitude change to a gravitational acceleration is small or if noise or drift occurring in the sensor is relatively large, particularly when a road surface is not a horizontal plane or a mobile body is inclined.
On the other hand, the method of using images of a surrounding of a mobile body which are captured by a camera mounted on the mobile body can, in principle, be used to correctly calculate a motion of the mobile body even when a road surface is not a horizontal plane or the mobile body is inclined.
As a representative method of obtaining a motion of a camera using captured moving images, a method of using optical flow has been proposed. Optical flow is a vector which connects points on two temporally consecutive images, the two points corresponding to a certain one point on an object. A geometric constraint expression is established between the corresponding points and the camera motion. Therefore, when a plurality of corresponding points satisfy a predetermined condition, the camera motion can be calculated.
For example, Nonpatent Publication 1 proposes a method called an eight-point method, in which a camera motion is calculated using a set of 8 or more corresponding static points between two images. However, in the eight-point method, a result of calculation of a corresponding point includes an error. Therefore, there are widely known problems. For example, it is difficult to extract 8 or more corresponding points with a small error. In addition, the instability of an expression for deriving a camera motion leads to a large error in calculation of the camera motion even if corresponding points have a relatively small error.
When a motion of a camera is calculated using actually captured images as with the eight-point method, it is substantially impossible to eliminate a measurement error, such as a large error (like an abnormal value), noise in the images, a quantization error or the like, which occurs in image processing.
Therefore, various techniques have been proposed in order to suppress an influence of an error in the method of calculating a camera motion using images. Examples of the techniques include a method of calculating a camera motion from a smaller number of corresponding points where a constraint condition is provided between corresponding points, a method of calculating a camera motion where the number of unknown parameters is reduced in the calculation of the camera motion, a method of calculating an unknown parameter in multiple stages, a method of using a filter such as a Kalman filter or the like, and the like.
For example, Nonpatent Publication 2 and Patent Publication 1 propose a method of calculating expressions of a camera motion and a plane from four or more corresponding points where all input corresponding points are assumed to be present on the same plane. This method is an example of a method of calculating a camera motion from less than 8 corresponding points, where a plurality of corresponding points are assumed to satisfy a certain constraint condition.
If the number of corresponding points which are used in calculation of a camera motion is small, it can be expected that a probability that a calculated camera motion contains an error is reduced when the corresponding points contain an error with a certain probability. When more than four corresponding points are used, it can be expected that by calculating a camera motion using a least square error method, an influence of a measurement error contained in a corresponding point can be reduced.
For example, Patent Publication 2 proposes a method of calculating a camera motion with a small error, in which a camera motion which is calculated from selected corresponding points on a road surface plane is subjected to a Kalman filter which is prepared taking into consideration a behavior of a vehicle or a position of a camera. By subjecting a calculated camera motion to a filtering process, an error which occurs in the camera motion calculation can be reduced.
For example, Patent Publication 3 and Nonpatent Publication 3 propose a method of calculating a camera motion using a search-like technique, in which the camera motion is limited to three parameters, i.e., translation and rotation on a plane, and a geometric constraint expression between corresponding points and the camera motion and an optical constraint expression relevant to values of pixels between small regions in the vicinity of corresponding points, are used. By placing a limitation on a camera motion to reduce the number of unknown parameters, it is possible to reduce an error in camera motion calculation.
For example, Nonpatent Publication 4 proposes a method of calculating a camera motion, in which unknown parameters which are required to calculate a camera motion from corresponding points are divided into two groups which are calculated in two respective stages relating to rotation or translation, whereby the number of unknown parameters to be calculated in one stage is reduced so that an error in camera motion calculation is reduced.
(Patent Publication 1) International Publication WO97/35161
(Patent Publication 2) Japanese Patent Laid-Open Publication No. 2001-266160
(Patent Publication 3) Japanese National Phase PCT Laid-Open Publication No. 2003-515827
(Nonpatent Publication 1) “A computer algorithm for reconstructing a scene from two projections.”, H. C. Longuet-Higgins, Nature, 293:133-135, September 1981
(Nonpatent Publication 2) “Gazorikai [Image Understanding]” Kenichi Kanetani, Morikita Shuppan, 1990
(Nonpatent Publication 3) “A Robust Method for Computing Vehicle Ego-motion”, G. P. Stein, O. Mano and A. Shashua, IEEE Intelligent Vehicles Symposium (IV2000), October 2000
(Nonpatent Publication 4) “Recovery of Ego-Motion Using Image Stabilization”, M. Irani, B. Rousso and S. Peleg, ICVPR, pp 454-460, March, 1994